This study presents a comprehensive evaluation of five leading large language models (LLMs) - Chat GPT 4o, Copilot Pro, Gemini Advanced, Claude Pro, and Meta AI - on their performance in solving calculus differentiation problems. The investigation assessed these models across 13 fundamental problem types, employing a systematic cross-evaluation framework where each model solved problems generated by all models. Results revealed significant performance disparities, with Chat GPT 4o achieving the highest success rate (94.71%), followed by Claude Pro (85.74%), Gemini Advanced (84.42%), Copilot Pro (76.30%), and Meta AI (56.75%). All models excelled at procedural differentiation tasks but showed varying limitations with conceptual understanding and algebraic manipulation. Notably, problems involving increasing/decreasing intervals and optimization word problems proved most challenging across all models. The cross-evaluation matrix revealed that Claude Pro generated the most difficult problems, suggesting distinct capabilities between problem generation and problem-solving. These findings have significant implications for educational applications, highlighting both the potential and limitations of LLMs as calculus learning tools. While they demonstrate impressive procedural capabilities, their conceptual understanding remains limited compared to human mathematical reasoning, emphasizing the continued importance of human instruction for developing deeper mathematical comprehension.

Under no circumstances should you let AI do your schoolwork for you, but Google has decided to make that option a little bit easier for the next year. The company is offering a free year of it's Google One AI Premium plan, which includes Gemini Advanced, access to the AI assistant in the Google Workspace and things like Gemini Live, to any college student willing to sign up. The offer gives you a sample platter of Google's latest AI features, which normally costs 20 per month, and is primarily focused on things you can do with Gemini. That includes experimental products like NotebookLM for analyzing documents, and Whisk for remixing images and videos. Because this is a Google One plan, you'll also get 2TB of Google Drive storage for the parade of PDFs that make up college life.

Abstract: This paper presents an in-depth analysis of the performance of seven different Large Language Models (LLMs) in solving a diverse set of math advanced calculus problems. The study aims to evaluate these models' accuracy, reliability, and problem-solving capabilities, including ChatGPT 4o, Gemini Advanced with 1.5 Pro, Copilot Pro, Claude 3.5 Sonnet, Meta AI, Mistral AI, and Perplexity. The assessment was conducted through a series of thirty-two test problems, encompassing a total of 320 points. The problems covered a wide range of topics, from vector calculations and geometric interpretations to integral evaluations and optimization tasks. The results highlight significant trends and patterns in the models' performance, revealing both their strengths and weaknesses - for instance, models like ChatGPT 4o and Mistral AI demonstrated consistent accuracy across various problem types, indicating their robustness and reliability in mathematical problem-solving, while models such as Gemini Advanced with 1.5 Pro and Meta AI exhibited specific weaknesses, particularly in complex problems involving integrals and optimization, suggesting areas for targeted improvements. The study also underscores the importance of re-prompting in achieving accurate solutions, as seen in several instances where models initially provided incorrect answers but corrected them upon re-prompting. Overall, this research provides valuable insights into the current capabilities and limitations of LLMs in the domain of math calculus, with the detailed analysis of each model's performance on specific problems offering a comprehensive understanding of their strengths and areas for improvement, contributing to the ongoing development and refinement of LLM technology. The findings are particularly relevant for educators, researchers, and developers seeking to leverage LLMs for educational and practical applications in mathematics.

Google's thoughtful AI research partner, Deep Research, is now available to Google Workspace users, Google said Thursday. And that's not all: Google Workspace users who sign up for Gemini Advanced can try other, experimental AI models, too. On Tuesday, Google announced that Gemini Advanced with Deep Research is now accessible by mobile users. On Thursday, Google migrated Gemini Advanced with Deep Research to Workspace users as well, provided that the Workspace subscription includes the extra Gemini Advanced subscription. Before this, Deep Research was only available in Gemini Advanced on the web, for 20 per month. Google's Workspace blog suggests that a salesperson could use Gemini Advanced with Deep Research to prepare a report on a prospective client, or a teacher could use it with lesson planning -- similar to what Google said Deep Research could do when it was originally announced.

Google is making Gemini just a bit better. Starting today, the company's chatbot will recall past conversations in an effort to provide more useful responses. "That means no more starting over from scratch or having to search for a previous conversation thread," Google explains. "Plus, you can build on top of previous conversations or projects you've already started." Google notes Gemini "may" indicate if it referenced a past conversation to formulate a response.

Chat GPT is not the only AI service in town. Google Gemini is a similar service where you can ask questions and get answers in plain text–no commands required. You can "converse" just as if the AI robot were a real person. If you're familiar with Chat GPT, you'll recognize it because the layout is similar. You're greeted by a stripped-down screen with a text input field at the bottom.

We evaluate the reliability of two chatbots, ChatGPT (4o and o1-preview versions), and Gemini Advanced, in providing references on financial literature and employing novel methodologies. Alongside the conventional binary approach commonly used in the literature, we developed a nonbinary approach and a recency measure to assess how hallucination rates vary with how recent a topic is. After analyzing 150 citations, ChatGPT-4o had a hallucination rate of 20.0% (95% CI, 13.6%-26.4%), while the o1-preview had a hallucination rate of 21.3% (95% CI, 14.8%-27.9%). In contrast, Gemini Advanced exhibited higher hallucination rates: 76.7% (95% CI, 69.9%-83.4%). While hallucination rates increased for more recent topics, this trend was not statistically significant for Gemini Advanced. These findings emphasize the importance of verifying chatbot-provided references, particularly in rapidly evolving fields.

As large language models (LLMs) are deployed in high-stakes domains like healthcare, understanding how well their decision-making aligns with human preferences and values becomes crucial, especially when we recognize that there is no single gold standard for these preferences. This paper applies a systematic methodology for evaluating preference alignment in LLMs on categorical decision-making with medical triage as a domain-specific use case. It also measures how effectively an alignment procedure will change the alignment of a specific model. Key to this methodology is a novel simple measure, the Alignment Compliance Index (ACI), that quantifies how effectively a LLM can be aligned to a given preference function or gold standard. Since the ACI measures the effect rather than the process of alignment, it is applicable to alignment methods beyond the in-context learning used in this study. Using a dataset of simulated patient pairs, three frontier LLMs (GPT4o, Claude 3.5 Sonnet, and Gemini Advanced) were assessed on their ability to make triage decisions consistent with an expert clinician's preferences. The models' performance before and after alignment attempts was evaluated using various prompting strategies. The results reveal significant variability in alignment effectiveness across models and alignment approaches. Notably, models that performed well, as measured by ACI, pre-alignment sometimes degraded post-alignment, and small changes in the target preference function led to large shifts in model rankings. The implicit ethical principles, as understood by humans, underlying the LLMs' decisions were also explored through targeted questioning. This study motivates the use of a practical set of methods and the ACI, in the near term, to understand the correspondence between the variety of human and LLM decision-making values in categorical decision-making such as triage.

At the beginning of 2022, a simplistic word-guessing game took the world by storm and was further adapted to many languages beyond the original English version. In this paper, we examine the strategies of daily word-guessing game players that have evolved during a period of over two years. A survey gathered from 25% of frequent players reveals their strategies and motivations for continuing the daily journey. We also explore the capability of several popular open-access large language model systems and open-source models at comprehending and playing the game in two different languages. Results highlight the struggles of certain models to maintain correct guess length and generate repetitions, as well as hallucinations of non-existent words and inflections.

Google's generative AI tools are getting some of the boosts the company previewed at Google I/O. Starting this week, the company is rolling out the next-gen version of its Imagen image generator, which reintroduces the ability to generate AI people (after an embarrassing controversy earlier this year). Google's Gemini chatbot also adds Gems, the company's take on bots with custom instructions, similar to ChatGPT's custom GPTs. Google's Imagen 3 is the upgraded version of its image generator, coming to Gemini. The company says the next-gen AI model "sets a new standard for image quality" and is built with guardrails to avoid overcorrecting for diversity, like the bizarre historical AI images that went viral early this year.